Papers with filtering methods
Investigating Web Corpus Filtering Methods for Language Model Development in Japanese (2024.naacl-srw)
Copied to clipboard
| Challenge: | a high quality web corpus is essential for large language models to be developed . strong filtering methods can lead to lesser performance in downstream tasks . |
| Approach: | They build classifiers and language models that can process large amounts of corpora quickly enough for pretraining LLMs. |
| Outcome: | The proposed method is the most accurate and leads to lesser performance in downstream tasks. |
Effectively Aligning and Filtering Parallel Corpora under Sparse Data Conditions (2020.acl-srw)
Copied to clipboard
| Challenge: | Parallel corpora are key to developing good machine translation systems, but abundant parallel data is hard to come by for languages with a low number of speakers. |
| Approach: | They propose an unsupervised alignment method that can handle rich morphology by removing incorrect translations and segments containing extraneous data. |
| Outcome: | The proposed method maximizes the number of correctly translated segments in a corpus and minimises noise by removing incorrect translations and segments containing extraneous data. |
Building a Japanese Typo Dataset from Wikipedia’s Revision History (2020.acl-srw)
Copied to clipboard
| Challenge: | Typographical errors (typos) also occur in user generated content (UGC). |
| Approach: | They extract over half a million Japanese typo–correction pairs from Wikipedia’s revision history and combine character-based extraction rules, morphological analyzers to guess readings, and various filtering methods to address these challenges. |
| Outcome: | The proposed dataset extracts over half a million typo–correction pairs from Wikipedia’s revision history. |
Towards Data-driven Ontologies: a Filtering Approach using Keywords and Natural Language Constructs (2020.lrec-1)
Copied to clipboard
| Challenge: | Creating ontologies is an expensive task. |
| Approach: | They evaluate two commonly used methods, OpenIE and co-occurrences, and use them to generate ontologies from documents. |
| Outcome: | The proposed methods perform better on pizza and agriculture document sets than OpenIE and co-occurrences. |
Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation (2024.acl-long)
Copied to clipboard
| Challenge: | generative large language models (LLMs) have found their application in data augmentation tasks, where small numbers of text samples are paraphrased and then used to fine-tune downstream models. |
| Approach: | They propose to use taboo words, hints by previous outlier solutions, and chaining on previous outliest solutions to augment text datasets as part of instructions to LLMs augmenting text dataset. |
| Outcome: | The proposed methods increase diversity of generated texts, but performance is highest with hints. |
TASA: Deceiving Question Answering Models by Twin Answer Sentences Attack (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing adversarial models rely on keyword matching and ignore relevant contextual relations for answer prediction. |
| Approach: | They propose to use keyword matching to attack model with two biases that rely on a perturbed answer sentence and a distracting answer sentence to misguide model. |
| Outcome: | The proposed method produces fluent and grammatical adversarial contexts while maintaining gold answers. |